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Investigating student choices in performing 
higher-level comprehension tasks using TED 

Francesca Bianchi 1 and Ivana Marenzi 2 



Abstract. The current paper describes a first experiment in the use of TED talks and 
open tagging exercises to train higher-level comprehension skills, and of automatic 
logging of the student’s actions to investigate the student choices while performing 
analytical tasks. The experiment took advantage of an interactive learning platform 
- Learn Web - that integrates TED talk videos and transcripts and enriches them 
with tagging features and a data logging system. The data collected offered an 
answer to the following questions: Which of the three tasks was perceived by the 
students as more difficult? flow was each task faced by the students? How did 
the logs contribute to an understanding of the students’ approaches to the tasks? 
The experiment also suggested ideas for further development of Learn Web’s log 
features from a pedagogical and research perspective. 
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1. Introduction 

Learn Web 3 (Marenzi, 2014) is an educational platform designed for retrieving, 
sharing, commenting and analysing multimedia resources. This platform has recently 
integrated TED talks (Taibi et al., 20 1 5) and enhanced TED transcripts with interactive 
features (e.g. text selections, annotations, and deletions; Bianchi & Marenzi, 2015). 

This experiment tested the newly integrated features within an academic module 
on interpreting. The module trains core skills, which include active listening and 
memorization. Learn Web was thus used to give students exercises targeting higher- 
level comprehension skills, and to gain insight into the student choices while 
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performing the given tasks. The following research questions led the analysis of 
the data: Which of the three tasks was perceived by the students as more difficult? 
How was each task faced by the students? How did the logs contribute to an 
understanding of the students’ approaches to the tasks? 

Below is a brief outline of the conceptual models which informed the exercises and 
the analyses in the current experiment. 

Discourse comprehension, memorization and summarization are fundamental skills 
in interpreting (Pochhacker, 2003), and expert trainers, such as Moser-Mercer 
(2000), suggest that an interpreter’s training should start from there. According to 
van Dijk and Kintsch (1983), comprehension, memorization and summarization 
involve the creation ofthree types ofmental representations: surface representations, 
focusing on microstructures; textbase representation, containing macrostructural 
information; and situation model , integrating text information with experience of the 
world. Macrostructures are created by applying processes of deletion, generalization 
and construction. Elaborating on van Dijk and Kintsch’s (1983) model, Brown and 
Day (1983) suggest six macrorules for text summarization: (1) deleting unnecessary 
information, (2) deleting redundant information, (3) substituting a list of items 
with a superordinate word, (4) substituting a list of actions with a word expressing 
a superordinate event, (5) selecting a topic sentence, and (6) inventing a topic 
sentence if none are available. Strategies involving substitution and construction are 
not used by less experienced readers (Winograd, 1983). Focus on text structure, i.e. 
the textbase representation, and deep understanding of the intended meaning of the 
text are fundamental in consecutive interpreting (Hatim & Mason. 1997). 


2. Method 

The experiment involved 25 Italian natives, attending a module on consecutive 
interpreting in an MA curriculum for language specialists. The students - aged 21- 
23 (except for one student aged 47) - were B2+ or above of the CEFR in English. 

The students were given three tasks revolving around the same TED video. Task 
1 asked them to divide the first six minutes of the video into five sections and 
give them a title. This task involves the creation of a textbase representation and 
requires the students to verbalize it, thus gauging understanding and summary 
skills. Task 2 asked the students to analyse the same piece of text and distinguish 
key assertions from ancillary pieces of text. This task draws attention to and 
requires the application of one of the most basic reading and summary strategies, 
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i.e. deletion. Finally, Task 3 required the students to mark each clause in the first 
three minutes of the speech with a discourse function chosen from a given list. 
This task draws attention to the communicative function of discourse, and involves 
abilities at the level of textbase representation and situation model, not to mention 
metalinguistic abilities. For more details, see Bianchi and Marenzi (2015). 

The tasks were performed using Learn Web’s selection and annotation feature, 
pictured in Figure 1. 

Figure 1. The annotation window 



LearnWeb keeps track of the student’s selections, deselections and annotations 
by means of a logging system and provides the teacher with a detailed log and a 
simple log. 


3. Discussion 

Due to some technical issues, only 14 resources in Task 1,15 resources in Task 2, 
and 1 3 resources in Task 3 were suitable for analysis, along with the corresponding 
logs. The following paragraphs present the results, organized by research question. 

3.1 . Which of the three tasks was perceived 

by the students as more difficult? 


To answer this question we used the detailed log, and manually counted the 
cases where a student had deselected a piece of text and subsequently changed 
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the selection span or the annotation. The percentage of students who substantially 
changed at least one selection or tag was 29%, 7%, and 38% in Tasks 1, 2, and 3, 
respectively; the number of substantial changes recorded was, respectively, seven, 
one, and eight. These data suggest that Task 2 was considered much easier than the 
other ones. Tasks 1 and above all Task 3, on the other hand, posed problems to the 
students, with Task 3 recording a higher number of substantial changes across a 
higher percentage of students. 

These results quantify and reflect the students’ confidence in the given skill. 

3.2. How was each task faced by the students? 

In Task 1, almost all the annotations matched Brown and Day’s (1983 ) macrorule 6 
(i.e. inventing a topic sentence), and involved strategies of substitution and creation. 
Furthermore, the students’ annotations were structured as long phrases (47%), 
complete sentences (38%), or wh questions (15%). These techniques suggest good 
summary skills and are also suitable to support memorization (see e.g. Taboada & 
Guthries, 2006; Urlaub, 2012). 

In Task 2, the students’ unit of reference was the paragraph (47%), the sentence 
(20%), or a mix of the previous two (33%). Considering key elements entire 
paragraphs of a text, though theoretically possible in some contexts, is not the 
best approach in the current scenario. Furthermore, 13% of the students used 
inappropriate tags. 

In Task 3, 77% selected and tagged full sentences or paragraphs instead of clauses, 
thus showing little sensitivity for discourse functions and limited familiarity with 
this type of pragmatic task. The students’ deselections and changes of tag were 
primarily located towards the beginning of the text. These results suggest that 
Task 2 was actually the most difficult, while Task 1 the easiest. 

3.3. How did the logs contribute to an understanding 
of the students’ approaches to the tasks? 


The detailed log proved fundamental to: understand whether a student had 
hesitations while performing the task; see the order in which selections and 
annotation were made; and understand the beginning and end of selections when 
multiple selections and tags appeared in contiguous pieces of text. 
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Due to the same technical issues that led to discarding some of the students’ 
resources, the simple log could not be used in the current analyses. 


4. Conclusions 

The current paper has shown a possible use of Learn Web’s interactive features 
and logs to gauge higher-level comprehension skills through selection and tagging 
exercises and the students’ confidence in their own skills. 

The results suggest that a large number of the students in this group had good 
summarising abilities - attested by their annotations in Task 1, based almost 
exclusively on strategies of substitution and creation but little awareness of the 
strategies and skills they automatically apply in the reading process - attested by 
their generally unsuitable selections in Tasks 2 and 3. Furthermore, the limited 
number of changes, especially in Task 2, suggest that most of the students were not 
aware of their limited metacognitive and analytical abilities. 

The analyses were largely based on evidence that was visible in the students’ final 
products, i.e. in the annotated resources. However, the system’s detailed log played 
more than one important role, as summarized in Section 3.3. In particular, it gave 
the researchers access to the draft versions of the students work. 
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